Focus detection devices that detect focus through a method known as the phase difference detection method in the related art detect an extent of image subject image shift based upon the outputs from a plurality of light receiving elements arrayed in correspondence to each of micro-lenses disposed in a two-dimensional pattern. Patent literature 1, for instance, describes a focus detection device that detects contrast along a plurality of directions and selects an optimal focus detection direction among the plurality of directions based upon the detected contrast.